# 8k long text processing

BTLM-3B-8k-base is a 3-billion-parameter language model with an 8k context length, trained on the 627-billion-token SlimPajama dataset, delivering performance comparable to open-source 7-billion-parameter models.

MPT-30B is an open-source large language model trained by MosaicML, based on a decoder-only Transformer architecture, pre-trained on 1 trillion English text and code tokens, supporting an 8k context window and efficient inference.